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1 .  Uncertainty  first,  risk  second 

2.  Based  on  GAO  reports,  some  things  aren’t  reducing  risk 
•  TRL’s,  Design  Maturity,  EVM 

3.  Software  development  and  integration  are  being  ignored 

4.  DoD  Workforce  are  big  risk  takers  even  when  they  shouldn’t  be 

5.  Workforce  needs  risk  management  training  that  focuses  on 
doing  it  effectively  (Proficiency) 

•  Not  learning  about  likelihood,  consequence,  mitigation  methods,  software 
tools,  etc. 

6.  If  you’re  not  successful  at  mitigating  risks,  don’t  do  it. 


“I  have  never  let  my  schooling  interfere  with  my  education.” 

-  Mark  Twain 
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1 .  Facts  concerning  major  weapon  programs 

2.  Survey  of  the  DoD  workforce  on  risk  management 

3.  Risk  behavior  of  the  DoD  workforce 

4.  Monte  Carlo  modeling  of  risk 

5.  Interviews  with  DoD  and  Industry  Program  Managers 


“If  we  don’t  succeed,  we  run  the  risk  of  failure.”  -  J.  Danforth  Quayle 
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What  I’m  Not  Going  to  Talk  About 
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>  What  is  Risk? 

>  The  DOD  Risk  Management  Guidebook 

>  Risk  Mitigation 

>  DAU 
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Topics 
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1 .  Facts  concerning  major  weapon  programs 

•  technology  maturity  (TRL),  design  maturity,  software  maturity, 
earned  value  management  (EVM) 

2.  Survey  of  the  DoD  Workforce  on  risk  management 

•  feedback  on  how  RM  is  used  on  programs 

•  recommendations  from  survey  participants 

3.  Risk  behavior  of  the  DoD  workforce 

•  propensity  to  take  on  risk 

4.  Simple  Monte  Carlo  modeling  of  risk 

•  Impact  on  program  cost  and  schedule 

5.  Best  practices,  lessons  learned  and  new  approaches  to  risk 
management  from  DoD  and  Industry  Program  Managers 
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GAO  Reports  For  Major  Weapon  Systems 

2003-2011 


Let’s  look  backward  to  prepare  for  the  future 


>  9  years  of  GAO  reports 

>  e.g.  “Defense  Acquisitions:  Assessments  of  Major  Weapon  Programs”,  GAO-03-476  (May  2003) 


>  Example  report 
2  pages  per  yr. 


Common  Name  JLENS 


Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System  (JLENS) 


The  Amy*  JLENS  will  provide  over-the-horizon 
detection  and  tracking  of  land-attack  cruise  missiles 
and  other  targets.  The  Arn<y  Is  developing  JLENS  111 
two  spirals.  Spiral  I  Is  complete  and  served  as  a  test 
tied  to  demonstrate  the  concept.  Spiral  2  will  utilize 
two  aerostats  with  advanced  sensors  for 
surveillance  and  tracking,  as  well  as  mobile  mooring 
stations,  communication  payloads,  and  processing 
stations  JLENS  will  provide  surveillance  and 
engagement  support  to  other  systems,  such  as 
PAC-3.  SM-6.  and  MEADS.  We  assessed  Spiral  1 


Program  Performance  (fiscal  year  201 1  dollars  In  million*) 


$1,975.5  $2,154.7 

$4,520.7  $5,041.9 

$6566.9  $7,378.0 

$410432  $461,122 


According  to  program  officials.  JLENS  will  enter 
production  with  mature  technologies,  a  stable 
dc*4gn.  and  proven  production  processes.  The 
program  began  development  in  2005  with  only  one 
of  Its  five  critical  technologies  mature,  and  only 
two  of  the  four  current  critical  technologies  are 
mature.  The  desdgn  appears  stable,  but  the 
potential  for  design  changes  remains  until  the 
maturity  of  JLENS  components  have  been 
demonstrated.  In  September  2010.  an  aerostat 
accident  resulted  m  the  loss  of  one  of  the  JLENS 
platforms.  This  accident  and  other  system 
integration  challenges  are  expected  to  delay 
several  key  program  events.  Including  the 
production  decision.  TVelve  of  the  program*  15 
critical  manufacturing  processes  are  currenUy  in 
control.  The  JLENS  program  has  also  completed  a 
number  of  key  production  planning  activities, 
such  as  assessing  supplier  capabilities  and  risks. 


Attainment  of  Product  Knowledge 


CAaii-taasr  A-~_r>u  rr,*T. 


Common  Name;  JLENS 


JLENS  Program 


Technology  Maturity 

JLENS  entered  system  development  in  August  2005 
with  only  one  of  us  five  critical  technologies  mature. 
The  program  subsequently  combined  two  of  the 
critical  technologies— the  communications  payload 
and  the  proccstang  group — into  the  communications 
processing  group.  The  communications  processing 
group  and  platform  are  currently  mature.  The 
program  expects  to  demonstrate  the  fire  control 
radar  and  surveillance  radar  in  a  realistic 
environment  before  the  program  enters  production. 
Many  of  the  JLENS  radar  technologies  have  legacy 
components.  However,  sensor  software  items 
related  to  signal  processing,  timing .  and  control,  as 
well  as  element  measurement,  are  not  yet  mature. 
The  program  office  has  successfully  conducted  tests 
of  the  fire  control  radar  antenna  but  the  integration 
of  both  the  fire  control  radar  and  surveillance  radar 
components  In  the  program*  system  integration 
laboratory  has  yet  to  occur. 

Design  Maturity 

The  JLENS  design  appears  stable,  but  the  potential 
for  design  changes  will  remain  until  key  JLENS 
components  have  been  integrated  and  tested.  For 
example,  a  (list  (light  demonstration  of  the  aerostat 
was  successfully  conducted  in  August  2009.  but  the 
program  mast  still  complete'  a  senes  of  tests 
integrating  the  JIJvNS  mobile  mooring  station  with 
the  aerostaL  In  September  2010.  the  program 
experienced  the  loss  of  a  platform  following  an 
aerostat  accident.  The  program  Is  analyzing  the 
cause  of  the  accident,  as  well  as  other  system 
integration  Issues.  The  JLENS  program  has  received 
approval  to  transport  the  mobile  mooring  station 
without  armor,  which  mitigates  a  risk  the  program 
office  has  Identified  in  the  past 

Production  Maturity 

The  JLENS  program  projects  that  It  will  ruler 
production  with  all  15  of  its  critical  manufacturing 
process*  mature  and  stable.  According  to  the 
program  office,  12  of  the  program*  critical 
manufacturing  processes  are  currently  In  control 
Tile  JLENS  program  has  also  completed  a  number  of 
key  activities  that  are  essential  to  effective 
production  management,  including  updating  its 
manufacturing  plan  and  addressing  areas  such  as 
supplier  capabilities  and  risks,  cost,  quality  control, 
materials,  pmdueihility.  and  workforce  skills. 


Other  Program  Issue* 

The  JLENS  program  Is  working  to  address  several 
risks  that  could  affect  the  program*  cost,  schedule, 
and  performance.  First,  the  program  received  $32 
million  less  than  the  amount  requested  In  the 
President*  fiscal  year  2010  budget  If  additional 
funding  Is  not  provided  in  fiscal  year  2012.  the 
program  reports  it  will  not  be  able  to  procure  the 
equipment  to  field  an  Initial  operational  capability  by 
the  end  of  fiscal  year  2013.  Second,  due  to  the 
September  2010  aerostat  accident  and  subsequent 
loss  of  a  platform,  the  program  experts  several  key 
events,  including  the  start  of  production,  to  be 
delayed.  Third.  If  problems  occur  during  systems 
Integration  and  verification  tests,  the  program 
expects  that  cost  and  schedule  would  be  affected. 
Fourth,  tf  test  site  preparations  are  not  complete  by 
April  301 1.  then  the1  production  Umrllnc  could  be 
Jeopardized.  Finally,  the  program  could  also  be 
affected  by  alignment  with  the  Army*  Integrated  Air 
and  Missile  Defense  program.  As  part  of  the 
Integrated  strategy,  the  Army  extended  the  system 
development  and  demonstration  phase  by  12 
months.  The  JLENS  program  Is  waiting  approval  of  a 
new  acquisition  program  baseline  with  up* laird  cost 
and  schedule  estimates  that  reflect  this  change. 

Program  Office  Comments 

In  commenting  on  a  draft  of  this  aacc&ancnt.  the 
program  stated  that  the  Army  is  planning  to  request 
funds  in  its  fiscal  year  2012  budget  to  offset  the  fiscal 
year  2010  reduction.  The  program  also  reported 
experiencing  development  challenges  that  have 
caused  system  integration  delay  s,  and  schedule 
challenges  due  to  a  September  2010  aerostat 
accident  The  program  office  continues  to  work  on  a 
now  acquisition  program  baseline.  A  now  cost 
estimate  w  as  presented  to  the  Army  Cost  Review 
Hoard  in  July  3009.  The  estimate  will  be  updated 
biased  on  the  results  of  an  Army  review  and 
submission  of  the  President*  fiscal  year  2012 
budget.  A  revised  baseline  is  expected  to  be 
approved  in  the  thin!  quarter  of  fiscal  year  2011.  The 
-Army  provided  technical  comments,  w  hich  were 
incorporated  as  appropriate. 


“You  can  observe  a  lot  by  just  watching.”  -  Yogi  Berra 
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Disclaimer  About  Data  in  the 
GAO  Reports 
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>  Non-attribution  (please). 

>  The  data  in  the  write-ups  were  used  as  is  and  there  wasn’t  an 
attempt  to  fix  any  apparent  discrepancies. 

>  The  impact  of  changes  in  the  program  scope  were  hard  to  identify. 

>  Costs  were  adjusted  for  inflation. 

>  Hard  to  compare  one  program  to  another. 


Errors  using  inadequate  data  are  much  less  than  those  using  no  data  at  all. 

-  Charles  Babbage 
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Technology,  Design  &  Production 
_ Maturity  -  Program  B _ 


Year 

Program  Unit- 
Cost  Growth* 

Schedule 

Delay 

Technology 

Design 

Drawings 

Completed 

2003 

9% 

0% 

4  of  5  mature 

Stable 

77% 

2004 

22% 

8% 

Remaining  tech, 
has  not  matured 

Changes  ...  to  mostly 
attain  better  mfg’ing 
efficiencies... 

67% 

2005 

21% 

10% 

All  mature 

Released. 

All  released  but  12%  to  be 
changed. 

2006 

36% 

24% 

All  mature 

Released 

All  released  but  12%  to  be 
changed. 

2007 

34% 

26% 

All  mature 

Production  decision 
delayed 

82% 

2008 

169% 

63% 

All  Mature 

Redesigning 

70% 

2009 

168% 

63% 

All  Mature 

Stable 

94% 

2010 

178% 

63% 

All  Mature 

Design 

“will  continue  to  evolve” 

96% 

2011 

170% 

71% 

All  Mature 

“Still  evolving” 

- 

•  Mature  technology.  •  170%  Unit  Cost  Growth  from  2003  to  201 1 

•  Design  never  stabilized.  •  71%  Schedule  Delay 

•  %  Drawings  completed  dropped  from  100%  to  70% 


‘Adjusted 
for  inflation. 
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Technology,  Design  &  Production 
Maturity  -  Program  B 
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Some  Risk  Management  Tools 
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>  Technology  Readiness  Levels  (TRL) 

•  Measure  of  risk  in  technology  development  during  EMD. 

>  Design  Drawings 

•  How  the  design  is  maturing  during  EMD. 

>  Software  Maturity 

>  Earned  Value  Management  (EVM) 

•  Used  to  forecast  cost  over-runs  and  schedule  delays  &  to  enable 
better  management  of  the  program 


EMD  =  Engineering  and  Manufacturing  Development,  DODI  5000.02 


“The  bigger  the  real-life  problems,  the  greater  the  tendency  for  the  discipline  to  retreat  into  a  reassuring 
fantasy-land  of  abstract  theory  and  technical  manipulation.”  -  Tom  Naylor 
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Assessments  From  the  GAO  Reports 

-Technology  Readiness  Levels 

— 


Technology  Readiness  Levels  (TRL) 

>  R&D  Cost  Growth*  after  reaching  full  maturity: 

•  75%  over  8  years 

•  65%  over  6  years 

•  1 6%  over  7  years 

•  6.5%  over  4  years 

•  120%  over  3  years 

•  2  programs  never  reached  full  maturity;  1  program  not  counted 

>  Technology  maturity  was  not  a  good  indicator  of  successful 
engineering  and  manufacturing  development. 

>  Technical  maturity  does  not  guarantee  cost  or  schedule 
performance. 

>  Overall,  TRL’s  are  not  as  useful  as  we  claim  they  are.  They  give  us 
an  overly  optimistic  assessment  of  program  risk. 


*  Adjusted  for  inflation 

Donald  C.  McKeon,  PhD  Presented  at  the  9th  Annual  Acquisition  Research  Symposium,  May  17,  2012 
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Assessments  From  the  GAO  Reports 
_ -Design  Drawings _ 


Of  8  programs. .. 

>  4  had  significant  drops  in  “releasable” 
drawings 

•  1 00%  to  70%,  98%  to  80%, 

98%  to  90%,  100%  to  82% 

>  1  had  a  10%  drop  after  5  years  at  100% 


2002  2003  2004  2005  2006  2007  2008  2009  2010  2011 


>  2  had  continuous  design  development 

>  1  did  not  have  useful  data 

>  Lesson  learned  -  not  an  accurate  way  to 
measure  the  maturity  of  the  design. 

>  Quoted  values  are  very  suspect. 

>  BUT,  it’s  a  leading  indicator  of  future 
problems!  (About  1  year) 


“Prediction  is  very  difficult,  especially  about  the  future.”  -  Niels  Bohr 


Donald  C.  McKeon,  PhD 
Defense  Acquisition  University 


Presented  at  the  9th  Annual  Acquisition  Research  Symposium, 
May  16-17,  2012  in  Monterey,  CA. 


May  17,  2012 
Slide  12  of  35 


PIS1  |  Assessments  From  the  GAO  Reports 

-  Software  Risks 
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>  -37%  of  the  program  write-ups  did  not  include  software  at  all. 


>  Interesting  quotes  from  the  GAO  reports 

•  “Developing  the  complex  flight  software  subsystem  ...  has  already 
caused  multiple  delays,  and  DCMA  has  reported  that  the  remaining 
software  effort  will  likely  further  delay  the  launch  ...” 

•  “..  the  greatest  risk  ...  is  software  development  and  integration...” 

>  Lesson  learned  -  The  focus  of  the  write-ups  tends  to  be 
“hardware”  focused  and  the  risk  of  software  development  and 
integration  is  often  ignored.  We  have  little  insight  into  a  major  area 
of  program  risk. 
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Assessments  From  the  GAO  Reports 


Defense  Acquisition  University 


>  For  8  programs  over  9  years,  Earned  Value  was  only  mentioned 
twice: 

•  “The  program  office  has  had  reduced  insight  into  its  prime  contractor's 
work  progress  since  December  2004  because  it  has  not  received 
detailed  earned  value  cost  and_  schedule  data.  ” 

•  “The  Defense  Contract  Audit  Agency  has  identified  significant 
deficiencies  with  the  prime  contractors’  earned  value  management 
system  that  affects  the  Air  Force’s  ability  to  oversee  the  cost  aspects 
of  the  program.  ” 

>  Lesson  learned  -  If  EVM  is  being  used  on  the  8  programs  studied, 
it  was  not  identified  in  the  yearly  write-ups.  There  is  no  indication 
that  program  execution  is  being  influenced  by  EVM  data. 


“Experience  is  that  marvelous  thing  that  enables  you  to  recognize  a  mistake  when  you  make  it  again.” 

-  F.  P.  Jones 
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1 .  Facts  concerning  major  weapon  programs 

•  technology  maturity  (TRL),  design  maturity,  software  maturity, 
earned  value  management  (EVM) 

2.  Survey  of  the  DoD  Workforce  on  risk  management 

•  feedback  on  how  RM  is  used  on  programs 

•  recommendations  from  survey  participants 

3.  Risk  behavior  of  the  DoD  workforce 

•  propensity  to  take  on  risk 

4.  Simple  Monte  Carlo  modeling  of  risk 

•  Impact  on  program  cost  and  schedule 

5.  Best  practices,  lessons  learned  and  new  approaches  to  risk 
management  from  DoD  and  Industry  Program  Managers 
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Survey  of  DoD  Workforce  Members  - 

Decision  Making 


>  “How  much  are  you  willing  to  spend  to  reduce  the  likelihood  of  a 
risk  from  50%  to  10%  With  a  Consequence  of  $10,000?” 

>  Expected  savings  is  $4000. 
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Survey  of  DoD  Workforce  Members  - 
Importance  of  Activities 


>  “In  your  opinion,  how  important  are  the  following  activities  for 
successful  risk  management?” 

Analysis  and  assessment 
Cooperation  from  others 
Subject  Matter  Expert  (SME)  advice 
Mitigation  Planning 
Detailed  risk  management  plan 

Training 

Expertise  in  risk  management 
Adequate  staffing 
Simple  process 
Adequate  funding 
More  action,  less  'wait  and  see" 

Software  tool(s)  for  entering,  tracking,  planning,  etc 

Experience  in  DoD  acquisition 
Less  action,  more  "think  it  through" 

More  frequent  risk  review  boards 

0%  10%  20%  30%  40%  50%  60%  70%  80%  90%  100% 
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Survey  of  DoD  Workforce  Members 
Importance  of  Activities 


>  Theme  #1 

•  analysis  and  assessment  (97%) 

•  SME  advice  (95%) 

•  mitigation  planning  (93%) 

•  training  (82%) 

•  expertise  in  risk  management  (79%) 

>  This  is  doing  risk  management,  not  learning  about  likelihood, 
consequence,  the  risk  matrix,  the  DoD  Risk  Management 
Guidebook,  etc. 

>  Proficiency  of  doing  risk  management  and  mitigation. 


If  you  think  education  is  expensive,  try  ignorance. 

-  Derek  Bok 


Donald  C.  McKeon,  PhD 
Defense  Acquisition  University 


Presented  at  the  9th  Annual  Acquisition  Research  Symposium, 
May  16-17,  2012  in  Monterey,  CA. 


May  17,  2012 
Slide  18  of  35 


Topics 


Defense  Acquisition  University 


1 .  Facts  concerning  major  weapon  programs 

•  technology  maturity  (TRL),  design  maturity,  software  maturity, 
earned  value  management  (EVM) 

2.  Survey  of  the  DoD  Workforce  on  risk  management 

•  feedback  on  how  RM  is  used  on  programs 

•  recommendations  from  survey  participants 

3.  Risk  behavior  of  the  DoD  workforce 

•  propensity  to  take  on  risk 

4.  Simple  Monte  Carlo  modeling  of  risk 

•  Impact  on  program  cost  and  schedule 

5.  Best  practices,  lessons  learned  and  new  approaches  to  risk 
management  from  DoD  and  Industry  Program  Managers 


“I  didn't  think;  I  experimented.”  -  Wilhelm  Roentgen 
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Results  of  Risk  Experiment  #2- 
Rewards  From  Risk  Taking 


Did  not 
take  risk, 
5.1% 


Program  is  Over  Budget 
95%  Took  Risk 


Program  is  On  Budget 
64%  Took  Risk 


Did  not 
take  risk, 
11.0% 


Risk  experiment.  98  students  (3  classes). 

All  students  started  on  budget.  Simulated  5  months. 

Students  could  select  the  budgeted  monthly  amount,  or  could  take  risk. 

No  risk  =  $10.  If  they  took  risk,  50%  Chance  $8  &  50%  chance  of  $14 
Students  on  or  under  budget  could  select  a  small  prize  (lowest  chooses  first) 


Program  is  Under  Budget 
89%  Took  Risk 
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Topics 


Defense  Acquisition  University 


1 .  Facts  concerning  major  weapon  programs 

•  technology  maturity  (TRL),  design  maturity,  software  maturity, 
earned  value  management  (EVM) 

2.  Survey  of  the  DoD  Workforce  on  risk  management 

•  feedback  on  how  RM  is  used  on  programs 

•  recommendations  from  survey  participants 

3.  Risk  behavior  of  the  DoD  workforce 

•  propensity  to  take  on  risk 

4.  Simple  Monte  Carlo  modeling  of  risk 

•  Impact  on  program  cost  and  schedule 

5.  Best  practices,  lessons  learned  and  new  approaches  to  risk 
management  from  DoD  and  Industry  Program  Managers 


“The  great  tragedy  of  Science  -  the  slaying  of  a  beautiful  hypothesis  by  an  ugly  fact.” 

-  Thomas  Henry  Huxley 
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Clarification  of  “Risk” 


£AU 
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>  Don’s  consideration  for  this  briefing 

>  Uncertainty  in  Effort 

•  the  task  is  required  and  will  be  completed  in  the  future 

•  there  is  some  uncertainty  in  the  cost  and  time  to  complete 
(typically  <  10%) 

>  “Known”  Risk  (planned) 

•  risk  has  been  identified  at  the  start  of  a  program 

•  negative  outcome  may  occur  in  the  future,  risk  mitigation  plan  exists, 
active  mitigation  is  started  at  program  start,  cost  and  schedule  have 
been  planned  for 

>  “Execution”  Risk 

•  risk  has  not  been  identified  at  the  start  of  a  program 

•  often  the  result  of  actions  outside  of  the  program 

•  negative  outcome  may  occur  in  the  future 

•  risk  mitigation  plan  does  not  exist,  cost  and  schedule  have  not  been 
planned  for 
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Risk  Management  Modeling 
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>  Extremely  simple  “Monte  Carlo”  modeling  of  risk 

>  Simulates  5  years  during  TD  and  EMD 

>  Options: 

•  cost  increase  for  every  task 

•  cost  variation  for  every  task 

•  setting  a  limit  on  cost  uncertainty  (i.e.,  no  tasks  less  than  budget) 

•  probability  of  tasks  that  have  risk 

•  likelihood  &  consequence 

•  cost  to  mitigate  a  risk 

•  effectiveness  of  mitigation  efforts 

>  For  simplicity,  the  total  budget  is  set  to  1  and  the  total  plan  is  60 
months. 


“I  have  yet  to  see  any  problem,  however  complicated,  which,  when  you  looked  at  it  in  the 
right  way,  did  not  become  still  more  complicated.” 

-  Poul  Anderson 
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The  Impact  on  Variation  in  Task 
Completion  (aka  Uncertainty) 


>  Assume:  Normal  distribution  of  costs  (average=1 ) 

>  Win  some  -  lose  some 

>  On  average... 

>  Historical  data 

>  Similar  tasks 


>  Or,  assume  no  cost  under-runs 


Conditions  that  might  prevent  a  cost  under-run 

>  timing  of  work 

>  different  teams,  divisions,  organizations, 
companies, ... 

>  contract  terms  /  subcontracting 

>  different  management  objectives 

>  different  reward  systems 

>  poor  understanding  of  actual  status 

>  deferred  tasks 

>  “there’s  always  more  that  can  be  done” 

>  sub-optimization 
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Actual  Cost  With  Normal  Uncertainty 
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Actual  Cost  Without  Under-Runs 
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Conclusion  #1 
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>  When  there  is  uncertainty  in  the  effort  to  complete  tasks,  in 
practice  there  will  be  a  cost  over-runs. 

>  All  DoD  programs  have  ambitious  goals  and  there  is  always 
uncertainty  in  program  plans. 

>  Therefore,  ALL  DoD  programs  will  have  a  cost  over-run,  a 
schedule  slip  and  a  likely  reduction  in  “performance”* 

>  A  cost  over-run  and  schedule  slip  to  some  degree  is  “natural”. 


“If  there  is  a  50-50  chance  that  something  can  go  wrong, 
then  9  times  out  of  ten  it  will.”  -  Paul  Harvey 
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£AU  The  Addition  of  “Execution  Risks” 
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l  - ^ - 

>  The  risk  has  not  been  identified  at  the  start  of  a  program 

>  High  risk  assumed:  10%  of  tasks  have  risk,  likelihood  =  50%,  cost 
consequence  is  50%  of  task  cost 

>  Without  mitigation,  average  cost  increase  is  2.5%. 
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Should  You  Mitigate? 


£AU 
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>  If  you  mitigate  all  risks  at  the  level  of  the  expected  loss  (=  Likelihood  * 
$Consequence)... 

a.  If  all  mitigation  efforts  are  successful.  Same  answer:  2.5%  increase. 

b.  But  if  mitigation  efforts  are  only  50%  successful,  the  cost  increase  is  3.8% 

>  Break-even  point :  C%Mi,  =  PSUccess 

>  Take-Away:  If  you  aren’t  successful  at  risk  mitigation,  don’t  do  it. 

>  Also,  for  optimal  risk  reduction,  you  need  to  know  how  successful  you  are  at 
risk  mitigation  (i.e.,  you  need  data). 


C0/oMit  is  the  fraction  of  the  expected  loss  that  is  spent  on  risk  mitigation. 
^success 's  the  probability  of  successfully  mitigating  risks. 
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Conclusion  #2 
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>  Execution  Risk  =  The  risk  has  not  been  identified  at  the  start  of  a 
program 

>  All  execution  risks  will  lead  to  a  cost  increase  and  schedule 
delay,  either  due  to  mitigation  efforts  and/or  consequences. 

>  The  optimal  risk  mitigation  strategy  requires  knowledge  of  the 
mitigation  success  rate. 

>  You  can’t  “manage  the  risk”  for  free. 


•  Performance  might  be  reduced  capabilities  or  fewer  assets. 
•  “All”  really  means  “on  average” 
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Topics 


Defense  Acquisition  University 


1 .  Facts  concerning  major  weapon  programs 

•  technology  maturity  (TRL),  design  maturity,  software  maturity, 
earned  value  management  (EVM) 

2.  Survey  of  the  DoD  Workforce  on  risk  management 

•  feedback  on  how  RM  is  used  on  programs 

•  recommendations  from  survey  participants 

3.  Risk  behavior  of  the  DoD  workforce 

•  propensity  to  take  on  risk 

4.  Simple  Monte  Carlo  modeling  of  risk 

•  Impact  on  program  cost  and  schedule 

5.  Best  practices,  lessons  learned  and  new  approaches  to  risk 
management  from  DoD  and  Industry  Program  Managers 
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Risk  Management  -  Company  X 
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>  Highly  profitable  company  outside  of  the  defense  industry. 

>  Consumer  and  business  customers. 

>  An  integrator  of  hardware,  software/firmware,  and  IT. 

>  Size:  100  engineers  /  7  platforms 

>  Close  working  relationship  with  ODM’s  (Original  Design 
Manufacturers;  1  is  critically  important  to  business  success) 

>  They  do  not  spend  a  lot  of  time  on  formalized  risk  management 
and  reviews. 

>  May  brief  5-10  high  risks  at  “phase”  reviews.  Have  trigger  points. 

>  Brainstorm.  Balance.  Highly  empowered  teams. 

>  Know  from  experience.  Informal:  everyone  does  it  but  they  might 
not  be  consciously  doing  “risk  management” 

>  Management  by  walking  around. 

>  Business  trade-off.  Some  flexibility  in  spending. 
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Risk  Management  -  Consultant  W 
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>  Commercial  IT  Business  Solutions 

>  Business  IT  solutions 

>  Risk  to  CFO  -  company  may  fail 

>  Cannot  put  dollar  or  time  value  of  risk 

>  (1 )  Business  (2)  People  (3)  Technology  and  (4)  Process 

>  Client  maturity 

>  Industry  standard  10% 

>  Seller  understands  the  fine  print.  Buyer  doesn’t  know  the  fine 
print. 

•  Works  both  ways  for  Gov’t  and  contractor.  RFP  vs.  proposal. 

>  More  than  1  year,  error  in  time  and  cost  is  100%. 

>  A  1-day  delay  can  turn  into  a  2-week  slip 

•  e.g.,  a  delayed  meeting  might  delay  an  important  decision  until  the 
“decider”  is  available 
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Program  Manager  Z 
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>  Manages  via  Risk  Management. 

>  Effective  RM  avoids  future  problems  so  that  time  can  be  spent  on 
program  issues  rather  than  fighting  fires. 

>  Used  DoD  Risk  Management  Guidebook  as  the  basis  for  RM,  but 
tailored  it  for  their  needs. 

>  Two  benefits  for  the  Branch  Manager  are  getting  visibility  into  the 
program  and  forcing  PM’s  to  look  at  risk. 

>  Tried  to  add  “urgency”  as  third  element  but  too  complex. 

>  “Believes”  in  payoff. 

>  Trying  to  measure  the  value  of  RM. 

>  During  technology  development: 

•  No  baseline. 

•  All  high  risk. 

•  Little  consequence  of  failure:  “go  as  far  as  you  can” 
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Conclusions  /  Summary 
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>  Uncertainty  first,  risk  second. 

>  Based  on  GAO  reports 

•  TRL’s,  Design  Drawings  and  EVM  aren’t  reducing  risk 

•  Software  development  and  integration  are  being  ignored 

>  The  DoD  Workforce  are  big  risk  takers  even  when  they  shouldn’t  be. 

>  Workforce  needs  risk  management  training  that  focuses  on  doing  it 
effectively  (proficiency  training). 

>  If  you’re  not  successful  at  mitigating  risks,  don’t  do  it. 

>  Cost  increases,  schedule  delays  (and  often  performance  shortfalls) 
are  normal  outcomes  of  uncertainty  and  “execution  risks”. 

>  Reduce  cost  over-runs  by  reducing  uncertainty  in  plans. 

>  More  deliverables  between  CDR  and  IOT&E. 


“The  secret  of  getting  ahead  is  getting  started.  The  secret  of  getting  started  is  breaking  your 
complex  overwhelming  tasks  into  small  manageable  tasks,  and  then  starting  on  the  first  one.” 

-  Mark  Twain 
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